Adaptive compression by light level

ABSTRACT

A method for adapting display data forming an image for display on a display screen to a viewer involves monitoring, over time, light that may affect an eye of the viewer of the image, analysing (S82) information regarding the monitored light based on one or more predetermined parameters (which may be based on a predetermined model of reactions of a human eye to light or to changes in light), adjusting (S83) at least one display value (preferably not a luminance value) of at least some of the display data based on the analysis of the monitored light, compressing the adjusted display data, and sending (S84) it for display on the display screen. In one embodiment, the one or more predetermined parameters includes one or more colour thresholds and monitoring the light comprises monitoring relative levels of different colours in the monitored light.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. national stage application under 35 U.S.C. §371 of International Patent Application No. PCT/GB2018/051348, filed onMay 18, 2018, which claims the benefit of Great Britain PatentApplication No. 1708060.7 filed on May 19, 2017, the contents of each ofwhich are incorporated herein by reference in their entirety.

BACKGROUND

Current development of display technology, especially in virtualreality, aims to match the performance of computer displays to theperformance of the human eye. This includes increasing the dynamicrange, which gives improved contrast, range of colours, and frame rate.However, as more and more display data is required in more complicatedscenarios, and it is desirable to use wireless technology, rather thanwires, the amount of data to be sent becomes very large. Although, insome cases, the display data is compressed, before being transmitted,wirelessly or through wires, it would still be desirable to try tominimise the amount of data that actually needs to be sent.

Accordingly, the present invention tries to reduce the amount of displaydata that needs to be sent for display to try to mitigate the aboveproblems.

SUMMARY

Therefore, in a first aspect, the invention provides a method foradapting display data forming an image for display on a display screento a viewer, the method comprising:

-   -   monitoring, over time, light that may affect an eye of the        viewer of the image;    -   analysing information regarding the monitored light based on one        or more predetermined parameters;    -   adjusting at least one display value of at least some of the        display data based on the analysis of the monitored light; and    -   sending it for display on the display screen.

In an embodiment, the method further comprises compressing the adjusteddisplay data prior to sending the adjusted display data for display onthe display screen. Preferably, adjusting the at least one display valueexcludes adjusting a luminance value of the display data. The one ormore predetermined parameters are preferably based on a predeterminedmodel of reactions of a human eye to light or changes in light.

Monitoring the light may comprise monitoring light emitted or to beemitted by the display screen when the image is displayed thereon, forexample by determining a luminance of the display data being displayedor to be displayed on the display screen, and/or monitoring the lightmay comprise monitoring ambient light in the vicinity of the viewer.

In a preferred embodiment, analysing the information based on one ormore predetermined parameters comprises comparing a level of brightnessof the monitored light to one or more brightness thresholds. The one ormore thresholds preferably includes a first brightness threshold, andadjusting the at least one display value preferably comprises, when themonitored level of brightness is below the first brightness threshold,performing one or more of:

-   -   reducing colour depth of the display data;    -   reducing a frame rate of the display data;    -   reducing a luminance depth of the display data by a first        amount; and decreasing a resolution of at least a portion of the        display data.

In an embodiment, the one or more thresholds includes a secondbrightness threshold, and adjusting the at least one display valuecomprises, when the monitored level of brightness of the light is abovethe second brightness threshold, performing one or more of:

-   -   reducing colour depth of the display data;    -   reducing a frame rate of the display data;    -   reducing a luminance depth of the display data by a first        amount; and    -   decreasing a resolution of at least a portion of the display        data;    -   for a predetermined period of time after it is determined that        the monitored level of brightness has increased to a brightness        level above the second brightness threshold.

In another embodiment, the one or more thresholds includes a thirdbrightness threshold, and adjusting the at least one display valuecomprises, when the monitored level of brightness remains below thethird brightness threshold for a first predetermined period of time,performing one or more of:

-   -   reducing values of red and/or yellow colour components of the        display data;    -   reducing a frame rate of the display data;    -   reducing a luminance depth of the display data by a second        amount, being higher than the first amount;    -   decreasing a luminance range of at least a portion of the        display data.

Adjusting the at least one display value may optionally comprisereducing values of blue and/or green colour components of the displaydata when the monitored level of brightness remains below the thirdbrightness threshold for a second predetermined period of time, and/ormay comprise removing all colour data of the display data when themonitored level of brightness remains below the third brightnessthreshold for a third predetermined period of time.

Reducing a luminance depth of at least a portion of the display data mayoptionally comprise relatively reducing a luminance depth of a portionof the display data forming part of the image compared to a luminancedepth of a portion of the display data forming a different part of theimage.

In an embodiment, the method may further comprise receiving feedbackfrom an eye tracking sensor as to which part of the image is beingfocussed on by the viewer, and relatively reducing the luminance depthof the part of the image being focussed on by the viewer and relativelyincreasing the luminance depth of a different part of the image which isbeing viewed using peripheral vision of the viewer.

Optionally, the one or more thresholds includes a fourth brightnessthreshold, and the method further comprises comparing the monitoredlevel of brightness with recent previous levels of brightness andadjusting the at least one display value comprises, when the monitoredlevel of brightness has increased by at least a predetermined amountfrom the recent previous levels of brightness to a brightness levelabove the fourth brightness threshold, performing one or more of:

-   -   reducing values of red and/or yellow colour of the display data        to a reduced amount;    -   reducing a frame rate of the display data to a reduced frame        rate;    -   reducing a luminance depth of the display data to a reduced        luminance depth;    -   reducing a resolution of at least a portion of the display data        to a reduced resolution;    -   reducing colour depth of the display data;    -   for a third predetermined period of time.

In this case, adjusting the at least one display value may optionallycomprise performing one or more of:

-   -   increasing values of red and/or yellow colour components of the        display data from the reduced amount;    -   increasing the frame rate of the display data from the reduced        frame rate;    -   increasing a luminance depth of the display data from the        reduced luminance depth;    -   increasing a resolution of at least a portion of the display        data from the reduced resolution;    -   increasing colour depth of the display data;    -   when the monitored level of brightness remains above the fourth        brightness threshold for a fourth period of time.

Preferably, the one or more predetermined parameters may include one ormore colour thresholds and monitoring the light may comprise monitoringrelative levels of different colour components in the monitored light.Monitoring the light may further comprise determining a level of redcolour component in the monitored light relative to other colourcomponents in the monitored light, and wherein adjusting the at leastone display value comprises, when the determined level of red colourcomponent in the monitored light relative to other colour components inthe light is above a first red colour threshold, performing one or moreof:

-   -   reducing values of non-red colour component data of the display        data relative to values of red colour component data in the        display data;    -   increasing a frame rate of the display data;    -   increasing a luminance depth of at least a portion of the        display data;    -   decreasing colour depth of the display data.

Increasing a luminance depth of at least a portion of the display datapreferably optionally comprises relatively increasing a luminance depthof a portion of the display data forming part of the image compared to aluminance depth of a portion of the display data forming a differentpart of the image.

The method my further optionally include receiving feedback from an eyetracking sensor as to which part of the image is being focussed on bythe viewer, and relatively decreasing the luminance range of the part ofthe image being focussed on by the viewer and relatively increasing adifferent part of the image which is being viewed using peripheralvision of the viewer

Monitoring the light may comprise monitoring the light with a photodetector or with a camera.

The display screen may be at an augmented reality device or virtualreality headset.

In a second aspect, the invention provides an apparatus configured toperform method described above.

In a third aspect, the invention provides a system comprising anapparatus as described above and an augmented reality device or virtualreality headset comprising the display screen.

Thus, in an example embodiment, the invention provides a method oftaking light adaptation of the photoreceptors in a user's eyes intoaccount when generating and transmitting display data, comprising:

-   -   1. Determining recent light level    -   2. Amending generated display data to produce only the parts of        the final display data which are assumed to be visible depending        on the determined light level    -   3. Transmitting the amended display data for display    -   4. Displaying the amended display data

Amending the generated display data may involve removing elements suchas colour depth, fast movement, and details requiring high acuity,reducing the volume of data being transmitted across a limited-bandwidthconnection.

The light level may be determined from knowledge of the display databeing transmitted or having been transmitted, or it may be determinedfrom a light measurement device in the vicinity of the viewer whichdetects the overall luminance of the displayed display data plus ambientlight level and colour balance.

In some embodiments, control of the ambient light level prior to thebeginning of transmission of display data may be used in order toacclimatise the user's eyes to a lower light level of the intendeddisplay images. Similarly, to improve the viewer's experience, theambient light could be changed from a more conventional white light to ared light in order to allow the user's eyes to transition to a level ofScotopic vision without removing ambient light entirely. Total darknesscould subsequently be used for a shorter period to ensure total darkadaptation. Red light could also be used to enable illumination suitablefor Photopic vision while also maintaining some level of Scotopicvision.

This allows elements of the display data to be adjusted so only elementsvisible to the viewer are transmitted. The adjustments could be madecontinuously to track the overall luminance of the display image.

The continual adjustment to the display data to match the actualvisibility of the images presented to the viewer can be used to reducethe overall data volume transmitted. This in turn reduces the bandwidthdemands on the communication path and also save power

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be more fully described, by way ofexample, with reference to the drawings, of which:

FIG. 1 shows the changing sensitivity of different retinal receptorsduring dark adaptation;

FIG. 2 shows the changing sensitivity of different retinal receptorsduring light adaptation;

FIG. 3 shows one embodiment of a system according to the invention;

FIG. 4 shows an embodiment of the system with an external lightmeasurement device;

FIG. 5 shows an embodiment of the system which is contained in a singledevice;

FIG. 6 shows an embodiment of the system with an internal lightmeasurement device;

FIG. 7 shows an embodiment of the system which is contained in a largespace;

FIG. 8 shows a flow chart of a method according to an embodiment of theinvention;

FIG. 9 shows a more detailed example system, with a model eye;

FIG. 10 shows a more detailed example decision-making process;

FIG. 11 shows a second detailed example system using a multi-purposeprogrammable processor;

FIG. 12 shows several further detailed example decision-makingprocesses;

FIG. 13 shows an example system with a single threshold;

FIG. 14 shows the decision-making process associated with the embodimentin FIG. 11; and

FIG. 15 shows a frame with areas of contrasting luminance.

DETAILED DESCRIPTION OF THE DRAWINGS

There are two basic types of photoreceptors in the human eye: rods andcones. The cones are mostly active in normal lighting, such as daylightor ambient indoor lighting. This provides what is known as Photopicvision, which includes full colour range, responsiveness to small, fastmovements, and greatest sensitivity in the foveal area of the eye asthis is where the cones are most concentrated.

In low light level conditions, such as dim lighting or moonlight, therods are more active and Photopic vision gives way to Scotopic vision.This is characterised by monochrome vision, low responsiveness to small,fast movements, and greatest sensitivity in the area outside the fovea.

In red lighting conditions, both types of vision can be used, allowinghigh acuity from the cones while preserving the sensitivity of the rods.This also allows illumination to be provided while preserving thechemical changes that occur in the photoreceptors during the transitionbetween Photopic and Scotopic vision.

During the transition between the types of vision, the so-called Pukinjeeffect or “blue shift” may occur. As the eye adapts to darkness, thepeak luminance sensitivity of the human eye shifts towards theblue-green end of the spectrum. This effect introduces a variation incontrast under different levels of illumination. It occurs because thecones are sensitive to red, green and blue light but require higherlevels of illumination to stimulate them than the rods. The rods cannotdiscriminate colour but are more sensitive at low light levels thancones and have the greatest response to light in the blue-green end ofthe spectrum. As the light intensity reduces the cones are stimulatedless, washing out most of the perceived colour. However, the objectsemitting or reflecting blue-green wavelengths appear relatively brighterthan objects emitting or reflecting light at the red end of the spectrumdue to the sensitivity of the rods at the shorter (blue/green)wavelengths.

FIG. 1 is a graph showing the decrease in the intensity of lightrequired to activate the two primary photoreceptors in the human retina.The vertical axis shows the threshold intensity from a high threshold toa low threshold. This indicates the intensity of light required beforethe light is detected. The lower the threshold, the more sensitive thephotoreceptor is.

The horizontal axis shows the time spent in the dark in minutes, dividedinto units of ten minutes, which are indicated with vertical lines whichintersect the curves showing the sensitivity of the photoreceptors.

Of these curves, the solid line shows the sensitivity of the cones,which are used for colour vision. These photoreceptors are moresensitive in bright conditions, but they do not have a large range ofsensitivity and so are less active in low light conditions. Thesensitivity curve levels out after about ten minutes in darkness andthere is negligible further improvement in sensitivity.

The dotted line shows the sensitivity of the rods, which are used formonochrome and low-light vision. Naturally, they are capable of a muchhigher level of sensitivity, as is shown on the graph in FIG. 1 by thefact that this curve reaches a much lower thresholdintensity—representing a lower level of brightness, or luminance,required for activating the rods and therefore higher sensitivity—thanthe curve showing the sensitivity of the cones.

Although the rods are less sensitive than the cones after up to tenminutes in darkness, the level of luminance required to activate themcontinues to fall after the sensitivity of the cones has reached itsmaximum. The sensitivity of the rods does not level to a maximum untilafter approximately 30-40 minutes in darkness, as shown by the low levelof the threshold luminance for the rods at this point.

FIG. 2 shows the sensitivity of photoreceptors which have adapted to lowlight intensity as described in FIG. 1. Specifically, it shows theincreasing intensity of light required to activate the rods and cones:as described in FIG. 1, the higher the threshold intensity as shown onthe vertical axis, the greater the intensity of light required foractivation of the appropriate photoreceptor. The solid line shows thesensitivity of the cones and the dashed line shows the sensitivity ofthe rods.

An increase in luminance reverses the chemical changes generated as partof dark adaptation and will cause a change from Scotopic to Photopicvision in less than 5 minutes, regardless of the size of the increase,provided it is sufficient to activate the cones. The level of theluminance is shown on the horizontal axis. If the light is very bright,the user will be “dazzled” as his or her photoreceptors areoverstimulated; as such, during the light adaptation period thephotoreceptors will be less sensitive. This is shown by the sharpupwards curves shown in the Figure.

During light adaptation, the retinal sensitivity gained during theprocess of dark adaptation described in FIG. 1 is lost, but the colourvision and increased acuity associated with Photopic vision are gainedas the rods are inhibited and the cones are re-activated.

Red light is an exception to this pattern; red light at higher thanthreshold luminance will have a reduced impact on dark adaptation of therods but will stimulate the cones sufficiently for Photopic vision toreturn. This means that an eye which has been exposed to red light canbe assumed to be dark adapted as if it has been exposed to an extremelylow light level, so exposure to red light will significantly shorten thetime required for full dark adaptation.

As previously mentioned, FIG. 1 and FIG. 2 show chemical changes whichare predictable and can therefore be modelled. The present invention ispreferably based on such a model of the reactions of the eye to lightwithin the system.

FIG. 3 shows a virtual reality headset [31] connected to a host device[32], which may be a computing device, gaming station, etc. The virtualreality headset [31] incorporates two display panels [33], which may beembodied as a single panel split by optical elements. In use, onedisplay is presented to each of a viewer's eyes. The host device [32]generates image data for display on these panels [33] and transmits theimage data to the virtual reality headset [31].

The host device [32] may determine, as it generates image data fortransmission to the virtual reality headset [31], how bright the imageswill be. In general, if the images are below a certain threshold ofluminance, the host device [32] then begins timing how long theluminance of the images being transmitted remains below that threshold(similarly, if the initial image data is below the threshold ofluminance, the host device [32] would begin timing when the stream ofimage data is initialised). It might also immediately reduce colourdepth and resolution to a minimum as the user will not be able to detectimage detail.

If the luminance remains below the threshold for more than a period oftime, e.g. ten minutes, the host [32] could adjust the image data beingsent to reduce the number of shades of colour compared to the number inthe image data as generated; drop red and yellow components of the lighttransmitted to leave only the blue components in order to account forblue shift; and/or potentially finally drop the colour density entirelyto monochrome after a further, longer period of time, e.g. fortyminutes. Furthermore, the host [32] could also reduce the resolution ofthe image to a minimum after this time. If the colour depth andresolution were reduced to a minimum upon the luminance falling belowthe threshold, they might be restored to low levels after ten minutes inorder to account for dark adaptation. The process would then continue asdescribed.

Because of the changing sensitivity of different parts of the user'sretina, such loss of detail will not be visible and as a result lessdata can be transmitted without any decline in user experience.

Similarly, if the light level suddenly rises, for example there is aflash or the scene being viewed is otherwise suddenly lit, the host [32]can continue to send display data with very little colour information,or indeed detail in general, for a period of time commensurate to theperiod for which the luminance of the frames has been low. This ispossible without affecting the quality of the images perceived by a userbecause after a long period of exposure to low-luminance images theuser's photoreceptors will be at their most sensitive, as indicated bythe adaption model of the eye, and they will therefore be overstimulatedby a sudden increase in luminance, reducing his or her acuity andsensitivity to colour.

If the light level continues to be high the frame rate and colourquantisation may be gradually increased, again based on assumptionsregarding light adaption as shown in FIG. 1. This is necessary because auser will lose dark adaptation and his or her photoreceptors will nolonger be overstimulated.

The user could also be “pre-conditioned” by spending a known amount oftime in a darkened or red-lit environment prior to putting on thevirtual reality headset [31] and the display data transmitted fordisplay could then be amended from the beginning rather than waiting fortime to pass. The use of such a method could be indicated to the host[32] via an override signal, which will pre-configure an adaption model.

FIG. 4 shows a second possible system. As in FIG. 3, there is a hostdevice [32] connected to a headset [41], but in this example the headset[41] is a set of augmented reality glasses. As in the virtual realityheadset [31] described in FIG. 3, there are two display panels [43],each associated with one of the user's eyes, but in this example thedisplay panels are translucent, so if they are being viewed by a userthe adaptation of his or her photoreceptors will also be affected byambient light. Accordingly, there is an external light measurementdevice [42] attached to the glasses and connected to the host device[32] in order to measure such luminance.

The host device [32] may be a static computing device such as acomputer, gaming console, etc., or may be a mobile computing device suchas a smartphone or smartwatch. As previously described, it generatesimage data and transmits it to the augmented reality glasses [41] fordisplay.

The light measurement device [42] is connected to the host device [32].It acts as a sensor to detect the ambient light level, and transmits[44] this to the host device [32], which may then send data with areduced colour gamut or resolution generally, as previously described.In this case, the data received [44] from the light measurement device[42] would be used in the same way as data collected on the luminance ofthe frames being transmitted to the display panels [33] in the virtualreality headset [31] described in FIG. 3, possibly in combination withknowledge of the luminance of the generated frames as previouslydescribed.

Alternatively, a camera pointed at the display panels [43] could act asa light measurement device, as this would provide actual knowledge ofthe light being presented to a user's eyes. Such a camera could bemounted on, for example, an arm of the glasses, or on the user's facenear or even inside his or her eyes, for example as part of a contactlens.

FIG. 5 shows a system which is similar in operation to the embodimentshown in FIG. 4. In this case, however, there is no separate host device[32]. The entire system is contained in a single casing [51], forexample in a smartphone or other such mobile computing device. Thedevice contains a processor [53], which generates display data fordisplay on the integral display panel [52].

The device [51] also has an external light measurement device [54],similar to that described in FIG. 4. Since the device [51] will not beenclosed, ambient light will affect the dark adaptation of any user'seyes, and as in the system described in FIG. 4, the external lightmeasurement device [54] monitors the ambient luminance. Alternatively, acamera pointed at the display panel [52] could act as a lightmeasurement device, as previously described with reference to FIG. 4. Ineither case, the camera or light measurement device [54] transmits [55]this data to the processor [53], where it is used to adjust the imagedata transmitted to the display panel [52] as previously described.

Since the system is entirely internal, the connection between theprocessor [53] and the display panel [52] is less likely to have alimited bandwidth compared to the connection between the host device[32] and either the virtual reality headset [31] described in FIG. 3 orthe augmented reality glasses [41] described in FIG. 4. This means thatit is less important to reduce the volume of data transmitted to thedisplay panel [52], but nonetheless the bandwidth is unlikely to beinfinite and therefore there may still be benefits to reducing thevolume of data, including memory use and latency.

Similar systems are currently used to lower the brightness of thebacklight in internal displays such as those in mobile devices. However,this is a different type of system as it involves lowering thebrightness of the backlight only; the display data is unchanged.

FIG. 6 shows a further example of a system in which a virtual realityheadset [61] is connected to a host device [32]. As previouslydescribed, the virtual reality headset [61] incorporates two displaypanels [62], one of which is presented to each of the user's eyes, andis itself likely to be sealed against ambient light when in use.

In this embodiment, the virtual reality headset [61] incorporates aninternal light measurement device [63]. As previously described, this isconnected to the host device [32], but unlike the light measurementdevice [42] described in FIG. 4, it transmits data on the luminancewithin the virtual reality headset [61]. This will detect thecombination of ambient light level actually present within the virtualreality headset [61] and the display such that, for example, if thereare warning lights not controlled by the host [32], their use will stillbe detected. The light measurement device [63] then transmits [64] thisinformation to the host [32], which can apply appropriate adjustments tothe display data being transmitted to the display.

Alternatively, a camera pointed at the display panels [62] could act asa light measurement device, as previously described with reference toFIG. 4.

The use of this system also means that if it is not convenient todetermine the luminance of frames as they are transmitted from the host[32], for example due to limited processing power in the transmitter orif the data is strongly encrypted, the described methods can still beused to amend the data as previously described during generation.

FIG. 7 shows a system which is similar in operation to the system shownin FIG. 6 in that there is an enclosed area [61/71] containing a displaydevice [62/72] and a light measurement device [63/73], and the user(s)[76] is/are viewing the display device [62/72] from within the enclosedarea [61/71]. In this embodiment, the system is in a theatre setting, orsimilar environment such as a motion-simulator fairground ride.

The display device [72] is a screen at the front of the environment[71]. It is being viewed by multiple users in an audience [76]. Theactual light level in the enclosed space [71] is detected by a lightmeasurement device [73] and transmitted [75] to the host [74], andtherefore if there is a change in the light level not controlled by thehost [74] which is providing the display data for the display, such as amember of the audience [76] opening an exit door into a lit vestibule,the resulting changes in the audience's [76] dark adaptation can beaccounted for. Since the chemical changes to the photoreceptors of thehuman eye are largely uniform across the population, it is likely thatthe eyes of all members of the audience [76] will adjust to lower lightlevels at the same rate.

Furthermore, this embodiment could be used for a method in which thelight level is controlled in order to manipulate the user's or users'[76] dark adaptation prior to showing display data. In such a method,the enclosed area [71] is sealed against ambient light for a long period(up to forty minutes) prior to the beginning of transmission of displaydata by the host [74]. The display data could then from the beginning beamended as previously described. Optionally, but beneficially, theenclosed area [71] could be red-lit during this time in order to allowthe users [76] to see enough to move around. This is possible due to theeffects of red light on the photoreceptors described above, meaning thata user can move from red light to darkness and be assumed to beessentially dark adapted.

Similarly, if the nature of the content to be transmitted as displaydata is known—for example, it is a pre-generated video clip, or a feedfrom a camera that only produces a particular type of display data—thetime between the enclosed area [71] being sealed against ambient lightand the beginning of transmission of display data could be adjustedaccordingly. For example, if the audience [76] is to view a brightly-litscene from a film on the display device [72], very little time elapsesbefore display begins, but if the audience [76] is to view a projectionof the night sky, which will be dim and monochrome, the full fortyminutes may be allowed to elapse.

Naturally, similar methods could be used for a single user using avirtual-reality headset [61] such as that shown in FIG. 6, and similarlya user could spend time in an area sealed against ambient light beforeusing the virtual-reality headset [61] in order to allow his or her eyesto adjust to low light levels.

FIG. 8 shows a flowchart of a generic version of the process followed,which will be described with reference to the embodiment shown in FIG.3, with deviations for the other embodiments described as appropriate.

At Step S81, the next frame of display data is generated in the host[32] according to the operation of applications running on the host[32].

At Step S82, the current or most recent light level is determined. Inthe embodiment shown in FIG. 3, this means determining the light levelof the frame most recently sent for display, combined with the lightlevels of other recent frames sent for display, for example over thelast minute or the last 120 frames. As previously described, this maymean maintaining a timer for how long the frames have been below aspecified threshold luminance. If there have been no previous frames,the luminance may be determined as a pre-set default luminance.

In the embodiments shown in FIGS. 4-7, which include light measurementdevices [42, 54, 63, 73], the luminance may be determined using theluminance as detected by the light measurement devices [42, 54, 63, 73],likely combined with the recent light level in a similar way to theluminance of a transmitted frame, and the output from the lightmeasurement devices [42, 54, 63, 73] may be used in conjunction with theluminance of frames.

Step S83, the amendment of the display data, may be subsequent to thegeneration of the display data in that data may be generated and thenamended, perhaps improving the compression applied prior totransmission. Alternatively, the first two steps may be reversed so thatthe information generated at Step S82 on light level may be fed directlyinto the processes generating the display data such that onlyappropriate display data is generated at all, in which case Steps 81 and83 are combined.

In either case, at the end of Step S83 only the data which can beassumed to be most beneficial to any user viewing the images is preparedfor transmission. This removes redundant data prior to thecompression/transmission, improving efficiency.

For example:

If the time period since the determined luminance was above a threshold(“T”) is short (say, less than five minutes), the display data can bepresented at full colour depth and frame rate. If appropriate, fovealcompression could also be used to reduce the quality of peripheral areasof the display without affecting user experience (naturally, if manypeople are viewing the same image, as in the embodiment in FIG. 7,foveal compression would not be possible; it uses eye-tracking in orderto determine the point of focus of a user). This is because any userwill still be using Photopic vision and therefore will have high acuityand responsiveness to fast movement and will be able to see a full rangeof colour, and due to the distribution of photoreceptors the user'sfovea will be the most sensitive area of the retina.

As an addition to this example, if a particular frame has a luminancebelow a threshold it can be amended to have a lower resolution andtherefore less detail. This is possible because a user's eyes will notbe sensitive enough in low light to see such detail; only the rods inthe eye are being stimulated, and these have lower density in theretina, so cannot perceive the normal full light level resolution. Suchamendment may be carried out in the compression process throughapplication of lossy compression methods, or by a reduction in thedetail by, for example, downscaling in rendered images, especially incomputer-generated images. For example, small text need not be renderedclearly.

If T is approximately ten minutes, red and yellow colour componentscould be dropped, leaving only blue components in an image that isotherwise monochrome, and foveal compression should no longer be used.This is appropriate because the sensitivity of the cones will level to aminimum while the rods will no longer be saturated and will be enteringtheir working range. Thus, as previously described, the user will betransitioning between Photopic and Scotopic vision, at which point he orshe will lose the ability to see detailed differences in colour, but themonochrome perceived contrast will change due to the Pukinje effect.Furthermore, the perifoveal area of a user's retina will be becomingmore sensitive than the fovea due to the distribution of photoreceptors,hence the reduction in usefulness of foveal compression.

If T is approximately twenty minutes, the number of frames transmittedto the display could be reduced, along with their colour quantisation;at this point all colour information could be removed and the framescould be displayed as monochrome images. This is appropriate because anyuser will be using Scotopic vision and therefore will have low acuity,reduced sensitivity to small, fast movements, and monochrome vision.These reductions can increase for up to forty minutes, at which point nofurther amendments should be made as a user's rods will have reachedfull sensitivity.

Correspondingly, if the luminosity of the display data increasessuddenly—for example, by a factor of greater than 20% between twoframes—the host may then send lower-quality frames for, say, a secondeven if the luminosity then returns to its previous level. This ispossible because if the user is exposed to sudden very bright light heor she will be “dazzled” and will lose detailed vision for a shortperiod.

Such methods may lower the volume of data by applying efficient coding,even if compression is not used, as permitted by the changes inperception of detail and colour described with reference to FIG. 1.However, compression may also be applied at this stage to further reducethe volume of the data to be transmitted across a limited-bandwidthconnection to the display.

This transmission is then carried out at Step S84. If the data wascompressed and/or encrypted prior to transmission, it may be received bya display control device connected to the display device receiving thedata [33], which may perform decompression and decryption asappropriate. The amended display data is displayed at Step S85.

FIG. 9 shows a more detailed block diagram of a system, showing apossible embodiment including a model of reactions of an eye to changesin light. This model is based on assumptions regarding the eye of auser, as described in FIGS. 1 and 2, but operates independently of theactual presence or absence of a user.

The majority of the system is incorporated into the host [91], which isconnected to a display control device [92], which is in turn connectedto a display device [93]. This display device [93] may be part of avirtual-reality headset such as those shown in FIGS. 3 [33] and 6 [62],a set of augmented-reality glasses such as that shown in FIG. 4 [43], ahand-held device such as that shown in FIG. 5 [52], or a theatre such asthat shown in FIG. 7 [72], or any other display system as appropriate.Likewise, the display control device may be part of such a headset [31,61], pair of glasses [41], or device [51], or simply connected to adisplay device.

Among other components, the host [91] includes a GPU [94], whichgenerates display data for display on the display device [93]. Thisdisplay data is most likely to be in the form of frames or parts offrames, which are passed to an analysis module [95] when complete.Alternatively, it may be passed to the analysis module [95] and alsodirectly to the compression engine [97], in which case the analysismodule [95] will not pass the frame to the compression engine [97] onceanalysis is complete.

The analysis module [95] analyses the frame in order to extract theluminance and dominant or average colour of the frame. This allows thesystem to monitor the level and colour of light that will be emitted bythe display device [93]. The analysis module then transmits the frame onto a compression engine [97] and the luminance and colour information toa processor [96] which contains the model of reactions of an eye tochanges in light.

The model includes a set of thresholds in a comparison engine [99] towhich the luminance of the frame produced in the GPU [94] is compared.The thresholds [99] consist of at least a main threshold, whichindicates the luminance at which the “eye” should begin dark adaptation,and a “dazzle” threshold, which indicates a luminance so high that itwould overstimulate any user's photoreceptors.

There are also two clocks [910, 911], of which one [910] measures thelength of time for which the “eye” has been exposed to bright light andis known as the “Photopic” timer since it will allow the model tosimulate the time at which a real eye would be likely to have returnedto Photopic vision. The other [911] measures the length of time sincethe “eye” was last exposed to bright light and is known as the“Scotopic” timer as it will allow the model [96] to simulate the time atwhich a real eye would have adjusted to use Scotopic vision. In the sameway as there are many thresholds [99], there may be many Photopic andScotopic timers [910, 911], associated with different thresholds [99].In an embodiment where the exposure of a user to light can be controlledbefore he or she begins viewing display data, information regarding suchprior exposure could be used to initialise the timers [910, 911] atvalues other than zero. For example, if a user is required to remain ina dark area for twenty minutes prior to the display of images on thedisplay device [93], the Scotopic timer [911] could be initialised attwenty minutes and the system would behave accordingly.

The values of the timers [910, 911] are used to calculate the level ofdark adaptation and are used along with the results of comparison to thethresholds [99] as inputs for the three specific adaptation elements[912, 913, 914]. Of these, the first [912] indicates the acuity of the“eye”. This indicates the level of detail that will be perceptible, aswell as the eye's ability to see small, fast movements. The secondelement [913] indicates the colour perception of the “eye”, and thethird [914] indicates the focus area: specifically, whether there shouldbe a foveal focus area, in which the point of focus on an image ishigh-quality while the periphery can be neglected. For this purpose,there may be an input from an eye-tracking mechanism when the system isin use, which will be used as input to either the processor [96] or thecompression engine [97]. Without such a mechanism or in the absence ofdata, this element [914] could be omitted or ignored.

Together, the thresholds [99], clocks [910, 911], and adaptationelements [912, 913, 914] provide a selection of parameters based onwhich the monitored light may be analysed.

Naturally, as more research is carried out into the perceptioncapabilities of the human eye, such a model may be embellished andextended to add further adaptation elements [912, 913, 914] or to changethose listed.

Each element [912, 913, 914] outputs an instruction regarding itsparticular part of the display data: resolution and frame speed, colourrendering, and focus area respectively.

These are gathered by an instruction engine [915] and transmitted to thecompression engine [97].

The compression engine [97] compresses the display data received fromthe GPU [94], via the analysis engine [95], using the instructionsreceived from the processor [96]. For example, it might apply fovealcompression to lower the resolution of the periphery while keeping thepoint of focus on the image high-resolution or, similarly, reduce theluminance depth of the area around the point of focus while relativelyincreasing the luminance depth of the remainder of the frame; reduce thenumber of shades of colour preserved; and/or reduce the overallresolution of the image according to instructions from the Focus Area[914], Colour [913], and Acuity [912] elements respectively. Thisamendment of the display data may be applied alongside other compressionmethods.

The compressed data is transmitted to the display control device [92]and decompressed in a decompression engine [98] if other compressionmethods have been applied. The amendments to the display data willsurvive. The amended display data is then sent to the display device[93] for display.

There may also be a light measurement device [916] which detects ambientluminance, as described in FIGS. 4 [42], 5 [54], 6 [63], and 7 [73]. Itis connected to the analysis module [95] and allows ambient light to betaken into account in determining the behaviour of the “eye”. This is afurther method of monitoring light that may affect the eye of a user.Since it is optional it is outlined in dashed lines, but in someembodiments it may replace the frame as input to the analysis module[95].

FIG. 10 shows a more detailed example process for Step S82, which willbe described with reference to FIG. 9.

At Step S101, a frame is generated in the GPU [94], consisting of anumber of pixels represented, in this embodiment, as Red, Green, andBlue values which will ultimately be displayed from the display device[93] as light. This frame is passed to the analysis module [95], whereit is analysed at Step S102. Because there will be two types of analysisperformed, this step is shown in FIG. 10 as two steps which occursimultaneously: Step S102A and Step S102B.

At Step S102A, the analysis module [95] determines the luminance of theframe by, in this example, which uses RGB colour components, determiningthe magnitude of each colour component in each pixel and therefore howbright each pixel will be, then averaging this value across the frame.The result will be a single figure for the luminance of the framegenerated at Step S101.

At Step S102B, the analysis module [95] determines the average colour ofthe frame, using the RGB values across the whole frame. This will alsoresult in a single colour value for the colour component of the light,and may in fact be a simple binary value for whether the light is red ornot, since this is the determination that may be required later in thisexample process. However, it may range from the differences betweenwhite light produced by florescent lighting and white light produced bysunlight to the differences between two coloured lights, such as a redlight and a green light.

If there is a light measurement device [916], its input may be fed intothe analysis module [95] and analysed in the same way, though it islikely to be analysed through spectral analysis rather than analysis ofRGB values. Naturally, a form of spectral analysis may also be used on aframe. If both a frame and a light measurement device are used, theirvalues may be combined.

Once both steps are complete the results are sent to the processor [96]containing the model eye and the process proceeds to Step S103. At thisstep, the luminance determined at Step S102A is compared to the mainthreshold value in the comparison engine [99]. This will allow theprocessor [96] to determine whether it would be appropriate to amend thedisplay data being transmitted to the display [93] and the degree ofamendment. All references to a threshold value herein may be to one of anumber of threshold values. This is necessary to simulate the continuaof increasing and decreasing sensitivity of the photoreceptors in a realeye. However, only the “main” and “dazzle” thresholds previouslymentioned will be used herein. Other thresholds will operate in asimilar manner.

In a first example, the luminance of the frame is determined to be abovethe main threshold, so the process follows the branch to the left,beginning at “Yes”, to FIG. 10 b.

At Step S10 b 1, the comparison engine [99] determines whether theluminance determined at Step S102A is not only above the main thresholdbut also above the “dazzle” threshold. The “dazzle” threshold may changedepending on previous luminance, such that if previously the luminancehas been very low over, for example, the previous ten minutes, thedazzle threshold may also be relatively low as any user's photoreceptorswould be at their most sensitive and therefore easy to overstimulate.Correspondingly, if the luminance had been high the dazzle threshold mayalso be very high. The previously-mentioned example of the light beingconsidered to be dazzling if the luminance rises by 20% between twoframes is an example of such a flexible dazzle threshold.

In a first example, the luminance does exceed the dazzle threshold andtherefore the process follows the left-hand branch beginning at “Yes” toStep S10 b 2. The results of the comparisons indicate that the “eye's”photoreceptors are overstimulated, and the comparison engine [99] sendsa signal to this effect to the three adaptation elements [912, 913,914], which determine that the compression engine [97] should reduce theresolution, frame rate, and/or colour quantisation of the data and mayalso or instead increase compression without regard to the quality ofthe resulting frame. This will not affect any user's experience sincehis or her photoreceptors will be overstimulated, as indicated by themodel [96], so he or she will have low acuity and will be unable to seedetails, fast movement, and colour detail. The adaptation elements [912,913, 914] send signals to this effect to the instruction module [915],which passes the instruction on to the compression engine [97], whichcarries them out at Step S10 b 2.

Meanwhile, the process moves on to Step S10 b 4 and the processor [96]increments the Photopic timer [910] to indicate that the luminance isabove the main threshold and the “eye” should begin to lose any darkadaptation.

In a second example, the comparison engine [99] determines that theluminance does not exceed the dazzle threshold and the process followsthe right-hand branch beginning at “No” to Step S10 b 3. At this step,the model [96] uses the Photopic timer [910] to determine how long theluminance has been above the main threshold. If the value of thePhotopic timer [910] is not above a time threshold, the model [96]increments the timer [910] at Step S10 b 5. No other action should betaken as it is unlikely that a user's eyes would have returned toPhotopic vision.

If the Photopic timer [910] is above the time threshold, the “eye” haslost dark adaptation and the adaptation elements [912, 913, 914] sendappropriate signals to the instruction element [915], forminginstructions that the compression engine [97] should adapt the displaydata to have a high frame rate, colour quantisation, and/or resolutionat Step S10 b 6. These values may be the full frame rate, colourquantisation, and resolution of which the host [91] and display device[93] are capable, or they may be reduced from the highest possible levelbut to a lesser extent than if the luminance had been determined at StepS103 to be below the main threshold. For example, if the Photopic timer[910] has a value of 3 minutes and 5 seconds, the time threshold [99]used for comparison is 3 minutes, frames were being transmitted at 20frames per second, and the total frame rate of which the system iscapable is 64 frames per second, the frame rate may be raised to 30frames per second. The comparison engine [99] may also begin using ahigher threshold for comparison such that if, later, the Photopic timer[910] has a value of 5 minutes and 2 seconds and the new threshold is 5minutes, the frame rate may be raised again to the full 64 frames persecond. Similar methods may be used for colour depth and resolution asappropriate. This is necessary because a real eye is likely to haveadjusted to the higher luminance. It will therefore have high acuity andbe able to perceive colour differences.

Finally, at Step S10 b 7 the colour data produced at Step S102B is usedto determine whether the received or produced light is red. This may beaccording to a threshold of average “redness” or may require aparticular area or location in a frame to be red. In either case, if thelight is red—it has previously been determined to be brighter than thethreshold—the processor [96] will increment both the Scotopic [911] andthe Photopic [910] timers at Step S10 b 8. This is so that the model[96] can take into account the fact that red light enables a real eye todark adapt to a degree or maintain previous dark adaptation.

This does not affect the signals sent by the adaptation elements [912,913, 914] to the instruction element [915] and hence the instructionssent to the compression engine [97], as Photopic vision can be usedunder red light, and therefore the data should be amended as describedat Step S10 b 6, as if the light had not been coloured. However, themodel [96] should also maintain high sensitivity to future low-lightconditions, so the Scotopic timer [911] is also incremented in case theluminance falls below the main threshold in future.

If the true colour of the frames does not matter, the applicationsrunning on the host [91] which generate display data could generateframes in shades of red as opposed to the full RGB, meaning that theframes will be analysed as red light. This will allow dark adaptationwhile still showing the frames at full luminance, as previouslymentioned.

If the light is not red, this Scotopic timer [911] will be reset at StepS10 b 9; the exposure to brighter light will have reversed any darkadaptation by the model [96] and such adaptation must now begin again.

In a third example, the comparison engine [99] determines that theluminance is not above the main threshold. The process then movesthrough the branch beginning “No” to FIG. 10c . At Step S10 c 1, theprocessor [96] increments the Scotopic timer [911] to record the lengthof time for which the luminance of the frames displayed (together withany ambient luminance, if there is a light measurement device [916]) hasbeen below the main threshold.

At Step S10 c 2, the newly-incremented timer [911] is checked todetermine how long the luminance has been below the main threshold. Forexample, the threshold is 20 minutes and the Scotopic timer [911]indicates that the luminance has been below the main threshold for 25minutes. The comparison engine [99] sends a signal to the adaptationelements [912, 913, 914] indicating that the “eye” has adapted todarkness to the degree indicated by the threshold used, and the processwill move to the branch beginning at “Yes”. At Step S10 c 3 theadaptation elements [912, 913, 914] transmit appropriate signals to theinstruction element, such that the Acuity element [912] indicates thatacuity is low, the Colour element [913] indicates that colour perceptionis low, and the Focus Area element [914] indicates that the fovea is notmore sensitive than the periphery. Accordingly, the instruction element[915] instructs the compression engine [97] to adjust the display datatransmitted to the display device [93] to reduce the frame rate,resolution, and/or colour depth, for example by representing colourvalues using four-bit numbers rather than the maximum eight-bit numbers,which will reduce the number of shades of the colour displayed. Theseamendments will be applied alongside any other compression methods, butthe instructions may also include an instruction to stop using fovealcompression if this is in use, so the quality of the display data willbe uniform across the frame.

The result is a lower volume of data being transmitted to the displaydevice [93], but any user will not notice any reduction in quality dueto the characteristics of dark adaptation: the cones will have reachedpeak sensitivity and the user is using Scotopic vision, which ischaracterised by monochrome vision, low responsiveness to small, fastmovements, and greatest sensitivity in the area outside the fovea, asrepresented in the model eye [96].

If the determined luminance has been below the main threshold for alength of time that is lower than the time threshold—for example, thetime threshold is 10 minutes and the light level has been below the mainthreshold for only 5 minutes—the model [96] will not send anyinstructions to the compression engine [97] to amend the display data oramend it further.

FIG. 11 shows a second block diagram of a system which does not haveadaptation elements [912, 913, 914] as shown in FIG. 9, but is stillcapable of monitoring light, analysing the resulting information, andadjusting the values of the display data in frames, using a set ofthresholds and timers.

As in FIG. 9, there is an application [114] which produces frames ofdisplay data. This runs on a processor and the rendering of the displaydata may be carried out on a GPU. In any case, the application [114]passes frames to an analysis engine [115] such as that described in FIG.9, and to a compression engine [117]. The analysis engine [115] is inturn connected to a multi-purpose processor [116] which is programmed tostore four timers [119] and five thresholds [118], as well has having anintegrated memory [1110] which can store the history of various analysisvalues, such as previous luminances over a predetermined period of time.

The thresholds [118] are values stored in memory. The firstfour—Threshold1, Threshold2, Threshold3, and Threshold4—are luminancevalues which, in the examples given herein, are used as follows:

-   -   Threshold1 and Threshold3: Determining whether the luminance has        dropped to a low level    -   Threshold2 and Threshold4: Determining whether the luminance has        risen to a high level

The two thresholds in any pair may have the same value or differentvalues. Naturally, this is an example only and in other embodiments thenumber and arrangement of thresholds may be different.

The fifth threshold [118]—ThresholdR—is a colour value which is used todetermine whether the frame is predominantly red in colour.

The timers [119] are used to record the length of time for which theluminance has been above or below particular thresholds. In thisexample, Timers A, B, and C are associated with Threshold3 to determinethe period for which the luminance has been low. Timer D is associatedwith Threshold4 and used to determine the period for which the luminancehas been high. The timers [119] have internal thresholds which are notshown for simplicity but are used to determine whether the values of thetimers [119] are sufficiently high to change the behaviour of thesystem.

As for the thresholds[118], the number and arrangement of the timers[119] is an example only.

As is the case for the model of the eye in FIG. 9, the thresholds andtimers allow analysis of the light that will be emitted from the displaypanel according to parameters which are embodied by these thresholds andtimers.

The processor [116] is also connected to the compression engine [117],which adjusts the display data and compresses it, prior to sending it tothe display control device [112]. The display control device [112]contains a decompression engine [1111], which decompresses the displaydata but does not reverse the adjustments to the display data prior tosending it to the display panel [113] for display.

FIGS. 12a-d show further example processes, which will be described withreference to FIG. 11. Any of these methods may be used in anycombination. In particular, since the values of FIG. 11's Threshold1 andThreshold3 (for example) may be the same, the associated processes maybe used together, and likewise for any other group of two or morethresholds.

FIG. 12a shows a process which will cause the system to react to lowluminance regardless of the length of time for which the luminance islow. At Step S12 a 1 a frame is generated by the application [114] aspreviously described. When the frame is displayed, light correspondingto the display values in the frame will be emitted from the displaypanel [113] and may affect the eye of a viewer of the image in theframe.

The frame is then passed to the compression engine [117], and also tothe analysis module [115], where it is analysed at Step S12 a 2 todetermine its luminance: i.e. the amount of light that will be emittedby the display panel [113]. As previously mentioned, other informationregarding the frame may be analysed at this stage, but it is notrelevant to this process.

At Step S12 a 3, the determined luminance is passed to the processor[116] and compared to the value of Threshold1 [118], which is stored inthe processor [116], and the processor [116] determines whether thedetermined luminance threshold is higher or lower than Threshold1 [118].If it is higher, the process follows the branch to the left and noaction is taken; the frame is not amended unless some other processbased on another threshold [118] or other considerations is also beingused as elsewhere described.

Otherwise, the process follows the branch to the right and the displaydata is adjusted. In the system described in FIG. 11, this is carriedout in the compression engine [117] and, accordingly, the processor[116] transmits a signal to the compression engine [117] indicating thatthe light level was lower than Threshold1 [118]. This causes thecompression engine [117] to adjust the display data in one or more ways:

-   -   Reducing colour depth by reducing the number of bits allocated        to the storage and/or transmission of each primary colour;    -   Reducing the frame rate by sending a feedback signal to the        application [114] instructing it to produce fewer frames or        dropping some frames until it receives a contradictory signal;    -   Reducing the luminance depth in either the whole of the frame or        part of it, indicated by, for example, feedback from an        eye-tracking mechanism, by reducing the number of bits allocated        to the storage and/or transmission of the luminance value;        and/or    -   Reducing the resolution of the frame by allowing greater        corruption of detail during compression

Naturally, the example methods given here are only examples and do notlimit the scope of possible adjustments and methods.

FIG. 12b shows a process which will cause the system to react to aperiod of high luminance. As previously described, at Step S12 b 1 aframe is generated by the application [114] and transmitted to theanalysis engine [115], which determines its luminance at Step S12 b 2.It then passes the luminance value to the processor [116], whichcompares it to the thresholds [118]. For the purposes of this processonly one threshold will be considered: Threshold2. This may have thesame value as Threshold1 and, indeed, in some embodiments they may bethe same threshold, but for this purpose they will be describedseparately.

At Step S12 b 3, the processor [116] determines whether the luminance ofthe frame is above Threshold2. If not, the processor [116] takes noaction based on this threshold. If the luminance of the frame is aboveThreshold2, the processor [116] sends a signal indicating this fact tothe compression engine [117]. The compression engine [117] then adjuststhe display data in one or more ways:

-   -   Reducing colour depth as previously described    -   Reducing the frame rate as previously described    -   Reducing the luminance depth as previously described; and/or    -   Reducing the resolution of the frame as previously described

Threshold2 in this example is analogous to the “dazzle” thresholddescribed in FIG. 10, and therefore the magnitude of these changesshould be reduced once time has passed. Accordingly, at Step S12 b 5 atimer in the compression engine [117] sends an override signal to thecompression engine [117]. This will cause the adjusted display values toreturn to their normal levels at Step S12 b 6. Naturally, this functionmay also be performed by a timer in the processor [116].

FIG. 12c shows a process which will cause the system to react to aperiod of low luminance as determined by Timers A, B, and C, aspreviously mentioned. At Step S1 c 1 a frame is generated as previouslydescribed, and at S12 c 2 the analysis engine [115] determines itsluminance and passes this value to the processor [116].

At Step S12 c 3, the processor [116] compares the luminance value toThreshold3 [118], as well as the other thresholds [118], which will notbe further discussed in this Figure. If the luminance is aboveThreshold3 [118], the processor [116] takes no further action withrespect to that threshold. Otherwise, the value of Timer A [119] iscompared to its associated threshold to determine whether the luminancevalue has been below Threshold3 [118] for a predetermined period of timewhich has been chosen as being long enough that the system should react.

If not, the timer [119] is incremented according to its functionality,but no other action is taken.

If the timer [119] does have a high enough value, the processor [116]sends a signal to this effect to the compression engine [117],indicating that the luminance is below Threshold3 [118] and Timer A[119] has a high value. As a result, at Step S12 c 4 the compressionengine [117] adjusts display values in the display data of the framesent to it by the application [114]. These adjustments may include oneor more of:

-   -   Reducing the levels of red and/or yellow in the frame;    -   Reducing the frame rate as previously described;    -   Reducing the luminance depth as previously described;    -   Reducing the luminance range by limiting the maximum luminance:        i.e reducing the difference between the brightest and darkest        luminance values in the display data; and/or    -   Reducing the resolution of the display data as previously        described

The process may end there, or a second timer [119], Timer B, may be usedto allow for further adjustments after more time has passed. At Step S12c 5 the value of Timer B [119] is compared to its associated thresholdto determine if enough time has passed to perform additional adjustmentsto the display data; this will indicate that the luminance has beenbelow Threshold3 [118] for a second predetermined period of time.

If the value of Timer B [119] is not high enough, it is incremented butno other action is taken. Otherwise, the processor [116] sends a signalto the compression engine [117] to this effect and, as a result, thecompression engine [117] reduces the level of blue in the frame at StepS12 c 6.

Again, the process may end there or a third timer [119], Timer C, may beused to allow further adjustments after a third predetermined period. AtStep S12 c 7 the value of Timer C [119] is compared to its respectivethreshold and if the appropriate timer has passed the processor sends afurther signal to the compression engine [117], causing it to reduce thecolour depth generally still further at Step S12 c 8. This may result inchanging the colour profile of the frame to monochrome.

Otherwise, Timer C [119] is incremented and the processor takes nofurther action.

The timers [119] may increment independently such that as soon as thefirst frame is below Threshold3 [118] they all begin timing, or they mayoperate sequentially such that as soon as one timer [119] passes itsrespective threshold the next begins timing. They may all be reset tozero as soon as the luminance of a frame is above Threshold3 [118], ormore than one frame may be required.

FIG. 12d shows a process which will cause the system to react to a largeincrease in luminance and a longer period of high luminance asdetermined by Timer D.

At Step S12 d 1, a frame is generated by the application [114], aspreviously described. It is passed to the analysis engine [115], whereits luminance is determined at Step S12 d 2, also as previouslydescribed. The resulting value is then passed to the processor [116] andcompared to Threshold4 [118] at Step S12 d 3.

If the luminance is not above Threshold 4 [118], the processor [116]takes no further action and any adjustment or lack of adjustment to thedisplay data being made in the compression engine [117] is unaffected.If the luminance is above Threshold4 [118], the process follows thebranch to the left, to Step S12 d 4.

At Step S12 d 4, the processor [116] checks a luminance value of aprevious frame stored in the memory [1110] to determine the size of anyincrease in luminance. If there has been no increase or the increase isby less than a predetermined amount, the process follows the branch tothe left and no further action is taken as a result of comparison toThreshold4 [118]. However, if the luminance has increased by at leastthe predetermined amount from a previous luminance to reach a levelabove Threshold4 [118], the process follows the branch to the right, toStep S12 d 5.

At Step S12 d 5, the processor [116] sends a signal to the compressionengine, [117] such as those already described, in this case indicatingthat the luminance has increased by at least a predetermined amount to alevel above Threshold4 [118]. This causes the compression engine [117]to adjust display values of the display data comprising the frame. Forexample:

-   -   Reducing the level of the red and yellow components of the        display data;    -   Reducing the frame rate as previously described;    -   Reducing the luminance depth as previously described;    -   Reducing the resolution as previously described; and/or    -   Reducing the colour depth as previously described

The process may stop there, or it may be expanded by the inclusion of atimer [119]: in this embodiment, Timer D is used. This indicates howlong the luminance has been above Threshold4 [118] and Timer D [119] maytherefore start timing at 0 when the first frame with a higher luminancethan Threshold4 [118] is analysed. This expansion is useful becauseThreshold4 [118] is also analogous to the Dazzle threshold previouslydescribed, and therefore it is beneficial to return the display data toits original state after time has passed.

In this case, the determination at Step 12 d 4 may still involvechecking a value stored in the internal memory [1110] of the processor[116], but in this case the value may be an indication of whether whenthe luminance first passed Threshold4 [118] it had increased by apredetermined amount. In this case, the process still follows the branchto the right, but since presumably Step S12 d 5 will have been carriedout when the luminance first passed Threshold4 [118], as previouslydescribed, this step could be omitted and the process could insteadfollow the dashed line directly to Step S12 d 6.

At Step S12 d 6, the processor [116] checks the value of Timer D [119]to determine how long the luminance has been above Threshold4 [118]. Ifthe value of the timer [119] is below the threshold, not enough time haspassed and the process follows the branch to the right: no furtheraction is taken and the compression engine [117] continues to make theadjustments indicated at Step S12 d 5.

If the appropriate length of time has passed, the process follows thebranch to the left and at Step S12 d 7 the processor [116] sends asignal to the compression engine [117] indicating that it should amendits behaviour to:

-   -   Increase the levels of the red and yellow components of the        display data;    -   Increase the frame rate;    -   Increase the luminance depth;    -   Increase the resolution; and/or    -   Increase the colour depth

Thus either partially or fully reversing the effects of the initialsignal sent at Step S12 d 5.

A similar effect may be achieved by the processor [116] simply sending asignal to the compression engine [117] to stop making the adjustmentsindicated at Step S12 d 5 after a predetermined period of time, or thecompression engine [117] having an internal timer that sends an overridesignal after a predetermined period of time in a similar way to thatdescribed in FIG. 12 b.

FIG. 12e shows a final process by which the behaviour of the system isadjusted when the frame is predominantly red in colour and willtherefore cause the display panel to emit red light.

At Step S12 e 1, the level of red in the frame and therefore the lightthat will be emitted is determined in the analysis engine [115]. Thismay be an absolute value or one relative to other colours present in theframe. This value is then transmitted to the processor [116], where itis compared to ThresholdR [118] at Step S12 e 2. If the level of red inthe frame is not above ThresholdR [118], no action is taken as a resultof the colour of the light, though luminance may cause the processor[116] to signal the compression engine [117] as previously described.

Otherwise, the process follows the branch to the left, to Step S12 e 3and the processor [116] sends a signal to the compression engine [117]indicating that it should adjust the display values in the display dataof the frame by, for example:

-   -   Reducing the levels of non-red colour components in the frame;    -   Reducing the frame rate as previously described;    -   Reducing the luminance depth as previously described; and/or    -   Reducing the luminance range as previously described.

Any and all of these methods may be used together and the signals sentmay cancel one another out in any appropriate way according to theembodiment.

FIG. 13 shows a system comprising a host [131], a display control device[132], and a display device [133] similar to those described in FIGS. 9and 11, but using a very simple threshold comparison engine [137] whichincludes only one threshold [138]. This threshold is still a parameterwhich is used in the analysis of the light which will be emitted by thedisplay device [133].

Accordingly, inside the host [131], the application [134] which producesframes is connected to an analysis engine [135], which analyses framesto determine their luminance as previously described. The analysisengine [135] is in turn connected to the comparison engine [137].

The application [134] also sends frames to a compression engine [136]for compression prior to transmission to the display control device[132]. The comparison engine [137] is also connected to the compressionengine [136] to enable it to transmit instructions which will controlthe compression carried out by the compression engine [136].

As in the systems described in FIGS. 9 and 11, the compressed data istransmitted to the display control device [132], where it isdecompressed prior to display on the display device [133].

FIG. 14 shows the process followed in this simpler embodiment of thesystem.

At Step S141, the application [134] produces a frame of display data, aspreviously described. It is transmitted to the compression engine [136]for compression, and also to the analysis engine [135]. At Step S142,the frame is analysed in the analysis engine [135] to determine itsluminance. This is carried out in the same way as the analysis describedin FIG. 10, but in this embodiment the colour of the frame is notdetermined as it will not be required by the comparison engine [137]. Itthen forwards the results of the analysis to the model [137].

At Step S143, the luminance determined at Step S142 is compared to thethreshold value [138] stored in the comparison engine [137]. In thisembodiment, this is a single luminance value, and if the determinedluminance of the frame is higher than the threshold [138] the processfollows the branch to the left, beginning at “Yes”, otherwise it followsthe branch to the right, beginning at “No”.

At Step S14Y1, the frame has been determined to be brighter than thethreshold [138] stored in the comparison engine [137]. In thisembodiment, the model [137] therefore instructs the compression engine[136] that it should retain the colour balance and resolution of theoriginal display data.

In contrast, at Step S14N1, the frame has been determined to be dimmerthan the threshold [138]. As a result, in this embodiment the comparisonengine [137] transmits an instruction to the compression engine [136]that it should lower the colour depth transmitted to the display controldevice [132]. Because this means that frames are represented by lessinformation, the volume of data transmitted to the display controldevice [132] will be lower. The lost data will not be restored upondecompression, resulting in fewer shades of colour being displayed inthe final image.

Naturally, a similar crude model could be used to control the behaviourof the host [131] with respect to any of the other properties of thedisplay data, such as resolution, frame rate, etc., and it may also beused in reverse to detect a light level that is determined to beover-intense, such that the user's eyes will not adjust sufficiently tosee details even with time. Finally, a model or similar mechanism may beof a level of complexity between the levels shown here. For example,there may be a single threshold with one or two fixed timers, or theremay be multiple thresholds without timers.

The methods of the invention have been described with respect todifferences in luminance across time. However, a method based on theinvention may also be applied across space such thatdrastically-different levels of luminance within a frame may affect theencoding of that frame. An example is shown in FIG. 15.

FIG. 15 shows a frame [151] which mostly consists of an area of lowluminance [152] featuring some detailed objects [153]. There is also anarea of high luminance [154] which likewise features some detailedobjects [155]. Overall, this frame [151] might have a low averageluminance. However, in this case it would be appropriate to apply themethods of the invention to the two parts [152, 154] separately,treating the high-luminance area [154] as requiring high colour depth,good resolution, fast movement, etc. while the low-luminance area [152]can be encoded with low colour depth, resolution, and movement.

The details [155] in the high-luminance area [154] would therefore beprioritised over the details [153] in the low-luminance area [153].

Not only could the two areas be differently encoded, but an analysisengine such as those described above could carry out analysis on thehigh-luminance area only, as long as there is a significant differencebetween the two areas [152, 154]—for example, the luminance of thelow-luminance area [152] is less than 50% of the luminance of thehigh-luminance area [154].

These methods improve the coding of display data according toassumptions about the perceptions of a viewer, in order to reduce thevolume of data being transmitted across a limited-bandwidth connection.This will allow the system to transmit data more quickly and with lessrisk of data loss or interference.

Although particular embodiments have been described in detail above, itwill be appreciated that various changes, modifications and improvementscan be made by a person skilled in the art without departing from thescope of the present invention as defined in the claims. For example,hardware aspects may be implemented as software where appropriate andvice versa, and engines/modules which are described as separate may becombined into single engines/modules and vice versa. Functionality ofthe engines or other modules may be embodied in one or more hardwareprocessing device(s) e.g. processors and/or in one or more softwaremodules, or in any appropriate combination of hardware devices andsoftware modules. Furthermore, software instructions to implement thedescribed methods may be provided on a computer readable medium.

The invention claimed is:
 1. A method for adapting display data formingan image for display on a display screen to a viewer, the methodcomprising: monitoring, over time, light that may affect an eye of theviewer of the image; analysing information regarding the monitored lightbased on one or more predetermined parameters by comparing a level ofbrightness of the monitored light to one or more brightness thresholds,wherein the one or more brightness thresholds includes a firstbrightness threshold and a second brightness threshold; adjusting atleast one display value of at least some of the display data based onthe analysis of the information regarding the monitored light by, whenthe level of brightness is below the first brightness threshold,performing one or more of: reducing colour depth of the display data;reducing a frame rate of the display data; reducing a luminance depth ofthe display data by a first amount; or decreasing a resolution of atleast a portion of the display data; sending the adjusted display datafor display on the display screen; and subsequent to sending theadjusted display data, when the level of brightness of the light isabove the second brightness threshold, performing, for a predeterminedperiod of time after it is determined that the level of brightness hasincreased to a brightness level above the second brightness threshold,one or more of: reducing the colour depth of the adjusted display data;reducing the frame rate of the adjusted display data; reducing theluminance depth of the adjusted display data by the first amount; ordecreasing the resolution of at least the portion of the adjusteddisplay data.
 2. The method according to claim 1, wherein the one ormore brightness thresholds includes a third brightness threshold, andadjusting the at least one display value comprises, when the level ofbrightness remains below the third brightness threshold for a firstpredetermined period of time, performing one or more of: reducing valuesof red and/or yellow colour components of the display data; reducing theframe rate of the display data; reducing the luminance depth of thedisplay data by a second amount, being higher than the first amount; ordecreasing a luminance range of at least a portion of the display data.3. The method according to claim 2, wherein reducing the luminance depthcomprises relatively reducing a first luminance depth of a first portionof the display data forming part of the image compared to a secondluminance depth of a second portion of the display data forming adifferent part of the image.
 4. The method according to claim 3, furthercomprising receiving feedback from an eye tracking sensor as to whichpart of the image is being focused on by the viewer, and relativelyreducing the luminance depth of the part of the image being focused onby the viewer and relatively increasing the luminance depth of thedifferent part of the image which is being viewed using peripheralvision of the viewer.
 5. The method according to claim 2, wherein theone or more brightness thresholds includes a fourth brightnessthreshold, and the method further comprises comparing the level ofbrightness with recent previous levels of brightness and adjusting theat least one display value comprises, when the level of brightness hasincreased by at least a predetermined amount from the recent previouslevels of brightness to the brightness level above the fourth brightnessthreshold, performing, for a third period of time, one or more of:reducing the values of red and/or yellow colour components of thedisplay data to a reduced amount; reducing the frame rate of the displaydata to a reduced frame rate; reducing the luminance depth of thedisplay data to a reduced luminance depth; reducing the resolution of atleast the portion of the display data to a reduced resolution; orreducing the colour depth of the display data.
 6. The method accordingto claim 5, wherein adjusting the at least one display value comprisesperforming, when the level of brightness remains above the fourthbrightness threshold for a fourth period of time, one or more of:increasing the values of red and/or yellow colour components of thedisplay data from the reduced amount; increasing the frame rate of thedisplay data from the reduced frame rate; increasing the luminance depthof the display data from the reduced luminance depth; increasing theresolution of at least the portion of the display data from the reducedresolution; or increasing the colour depth of the display data.
 7. Themethod according to claim 2, wherein adjusting the at least one displayvalue comprises reducing values of blue and/or green colour componentsof the display data when the level of brightness remains below the thirdbrightness threshold for a second predetermined period of time, orwherein adjusting the at least one display value comprises removing allcolour data of the display data when the level of brightness remainsbelow the third brightness threshold for a third predetermined period oftime.
 8. A method for adapting display data forming an image for displayon a display screen to a viewer, the method comprising: monitoring, overtime, light that may affect an eye of the viewer of the image; analysinginformation regarding the monitored light based on one or morepredetermined parameters by comparing a level of brightness of themonitored light to one or more brightness thresholds, wherein the one ormore brightness thresholds includes a first brightness threshold and asecond brightness threshold; adjusting at least one display value of atleast some of the display data based on the analysis of the informationregarding the monitored light, wherein adjusting the at least onedisplay value comprises, when the level of brightness is below the firstbrightness threshold, removing one or more elements of the image andperforming one or more of: reducing colour depth of the display data;reducing a frame rate of the display data; reducing a luminance depth ofthe display data by a first amount; or decreasing a resolution of atleast a portion of the display data; and sending the adjusted displaydata for display on the display screen; and subsequent to sending theadjusted display data, when the level of brightness has increased abovethe second brightness threshold, further performing the one or more of:reducing the colour depth of the adjusted display data; reducing theframe rate of the adjusted display data; reducing the luminance depth ofthe adjusted display data by the first amount; or decreasing theresolution of the at least the portion of the adjusted display data. 9.The method according to claim 8, wherein adjusting the luminance depthof the display data comprises relatively reducing the luminance depth ofa first portion of the image in the display data as compared to a secondportion of the image.
 10. The method according to claim 1, wherein theone or more predetermined parameters are based on a predetermined modelof reactions of a human eye to light or changes in the light.
 11. Themethod according to claim 1, wherein monitoring the light comprisesmonitoring light emitted or to be emitted by the display screen when theimage is displayed thereon, wherein monitoring the light comprisesdetermining a luminance of the display data being displayed or to bedisplayed on the display screen.
 12. The method according to claim 1,wherein the one or more predetermined parameters includes one or morecolour thresholds and monitoring the light comprises monitoring relativelevels of different colour components in the monitored light.
 13. Themethod according to claim 1, further comprising compressing the adjusteddisplay data prior to sending the adjusted display data for display onthe display screen.
 14. The method of claim 1, wherein adjusting the atleast one display value of the at least some of the display data basedon the analysis of the information regarding the monitored lightcomprises adjusting a first portion of the display data to have a highercolour depth, a higher resolution, or a faster movement over a secondportion of the display data.
 15. A system comprising: a non-transitorymemory storing instructions; and one or more hardware processors coupledto the non-transitory memory and configured to execute the instructionsfrom the non-transitory memory to cause the system to perform operationscomprising: monitoring, over time, light that may affect an eye of aviewer of an image associated with display data for display on a displayscreen to the viewer; analysing information regarding the monitoredlight based on one or more predetermined parameters by comparing a levelof brightness of the monitored light to one or more brightnessthresholds, wherein the one or more brightness thresholds includes afirst brightness threshold and a second brightness threshold; adjustingat least one display value of at least some of the display data based onthe analysis of the information regarding the monitored light, whereinadjusting the at least one display value comprises, when the level ofbrightness is below the first brightness threshold, removing one or moreelements of the image and performing one or more of: reducing colourdepth of the display data; reducing a frame rate of the display data;reducing a luminance depth of the display data by a first amount; ordecreasing a resolution of at least a portion of the display data; andsending the adjusted display data for display on the display screen; andsubsequent to sending the adjusted display data, when the level ofbrightness has increased above the second brightness threshold, furtherperforming the one or more of: reducing the colour depth of the adjusteddisplay data; reducing the frame rate of the adjusted display data;reducing the luminance depth of the adjusted display data by the firstamount; or decreasing the resolution of the at least the portion of theadjusted display data.
 16. The system according to claim 15 and anaugmented reality device or virtual reality headset comprising thedisplay screen.
 17. A method for adapting display data forming an imagefor display on a display screen to a viewer, the method comprising:monitoring, over time, light that may affect an eye of the viewer of theimage; analysing information regarding the monitored light based on oneor more predetermined parameters; adjusting at least one display valueof at least some of the display data based on the analysis of theinformation regarding the monitored light; and sending the adjusteddisplay data for display on the display, wherein the one or morepredetermined parameters includes one or more colour thresholds, whereinmonitoring the light comprises determining a level of red colourcomponent in the monitored light relative to other colour components inthe monitored light, and wherein adjusting the at least one displayvalue comprises, when the determined level of red colour component inthe monitored light relative to other colour components in the light isabove a first red colour threshold, performing one or more of: reducingvalues of non-red colour component data of the display data relative tovalues of red colour component data in the display data; increasing aframe rate of the display data; increasing a luminance depth of at leasta portion of the display data; or decreasing colour depth of the displaydata.